🌴KapoeiraπŸ•ΊπŸ’ƒ

kapoeira

Presentation

kara
Mehdi Rebiai

Presentation

odile
Johanna Vauchel

🎬 Kapoeira story πŸ“½οΈ

Enrich and collect data

enrichData

We are perfect !

perfect

Data is perfect !

pipeline example

NO !

pipeline example poo

Solution ?

TESTS !

How to test ?

TopologyTestDriver

fast

Fast and efficient…​

Scala Test Example

…​ But it’s mock
fail


it did not test the integration with the Kafka cluster
…​ But unitary
end to end


how to test several streams ?
…​ But it’s not for QA
les nuls police


Listen, let’s DEV do their job !
…​ But it’s code
simon kara


how to communicate with DEV/PO/QA ?


What do we need ?
test pyramid
Integration tests with simple syntax
What do we need ?
cesar triangle


Integration tests with simple syntax
What do we need ?
kafkalogo


Integration with Kafka Streams
Inspiration : Karate
karate


  • HTTP-based APIs
  • Peter Thomas - 2017
  • simple syntax (Gherkin)


Birth of Kapoeira
2020
  • Inner tool
  • console-script confluent
202x ZIO
zio
2023 Open Source
kapoeira logo text


Inside Kapoeira
Cucumber Scala, using specific Gherkin DSL.
cucumber
How does it work ?
kapoeira diagram
Example
Feature: upper-case
  Background:
    Given input topic
      | topic              | alias    | key_type | value_type |
      | topic-simple-value | topic_in | string   | string     | (1)

    And output topic
      | topic                   | alias            | key_type | value_type | readTimeoutInSecond |
      | topic-simple-value      | topic_string_out | string   | string     | 5                   |
      | topic-upper-case-string | topic_out        | string   | string     | 5                   |  (2)
    And var myKey = call function: uuid

  Scenario: My first scenario
    When records with key and value are sent (3)
      | topic_alias | key      | value |
      | topic_in    | ${myKey} | a     |
      | topic_in    | ${myKey} | b     |
      | topic_in    | ${myKey} | c     |
    Then expected records                    (4)
      | topic_alias      | key      | value    |
      | topic_string_out | ${myKey} | input_1  |
      | topic_string_out | ${myKey} | input_2  |
      | topic_string_out | ${myKey} | input_3  |
      | topic_out        | ${myKey} | result_1 |
      | topic_out        | ${myKey} | result_2 |
      | topic_out        | ${myKey} | result_3 |
    And assert input_1 $ == "a"             (5)
    And assert input_2 $ == "b"
    And assert input_3 $ == "c"

    And assert result_1 $ == "A"
    And assert result_2 $ == "B"
    And assert result_3 $ == "C"
Report
report
REX
Advantages
advantages
Advantages
  • test infra
  • simple to use
  • tool to communicate
  • documentation
  • acceptance tests


How to use ?
banco


How to build ?
docker build -t kapoeira:latest .


How to use ?
Docker Hub image


docker run --rm -ti \
-v <PATH_TO_YOUR_FEATURES_FOLDER>:/features \
-v /var/run/docker.sock:/var/run/docker.sock \
-e KAFKA_BOOTSTRAP_SERVER=<HOST:PORT[,HOST2:PORT2,HOST3:PORT3,...]> \
-e KAFKA_SCHEMA_REGISTRY_URL=<URL> \
-e KAFKA_USER=<XXX> \
-e KAFKA_PASSWORD=<****> \
-e JAAS_AUTHENT=<true (default) | false> \
-e LOGGING_LEVEL=<INFO (default) | ERROR | ...> \
-e THREADS=<8 (default) | ... > \
lectratech/kapoeira


How to contribute ?
TODO


DΓ©mo
buger quiz
Architecture
burger quiz
Docker commands
docker compose build --no-cache
docker compose up -d
docker restart kapoeira
burger.feature
Feature: Burger πŸ” feature

  Background:
    Given input topic
      | topic     | alias        | key_type | value_type |
      | bread     | bread-in     | string   | string     |
      | vegetable | vegetable-in | string   | string     |
      | meat      | meat-in      | string   | string     |
    And output topic
      | topic  | alias      | key_type | value_type | readTimeoutInSecond |
      | burger | burger-out | string   | string     | 5                   |
    And var uuid = call function: uuid

  Scenario: Nominal
    When records with key and value are sent
      | topic_alias  | key        | value |
      | bread-in     | 🀀_${uuid} | 🍞    |
      | vegetable-in | 🀀_${uuid} | πŸ…    |
      | meat-in      | 🀀_${uuid} | πŸ₯©    |
    Then expected records
      | topic_alias | key        | value |
      | burger-out  | 🀀_${uuid} | order |
    And assert order $ == "πŸ”"

  Scenario: Not a burger
    When records with key and value are sent
      | topic_alias  | key        | value |
      | bread-in     | 🀀_${uuid} | 🍞    |
      | vegetable-in | 🀀_${uuid} | πŸ₯•    |
      | meat-in      | 🀀_${uuid} | πŸ₯©    |
    Then expected records
      | topic_alias | key        | value |
      | burger-out  | 🀀_${uuid} | order |
    And assert order $ == "🍞 + πŸ₯• + πŸ₯©"

  Scenario Outline: Many customers
    When records with key and value are sent
      | topic_alias  | key            | value       |
      | bread-in     | <user>_${uuid} | <bread>     |
      | vegetable-in | <user>_${uuid} | <vegetable> |
      | meat-in      | <user>_${uuid} | <meat>      |
    Then expected records
      | topic_alias | key            | value |
      | burger-out  | <user>_${uuid} | order |
    And assert order $ == "<result>"

    Examples:
      | user | bread | vegetable | meat | result |
      | 🀀   | 🍞    | πŸ…        | πŸ₯©   | πŸ”     |
      | πŸ˜‹   | 🍞    | πŸ…        | πŸ—   | πŸ”     |
      | 😑   | 🍞    | πŸ…        | 🐟   | πŸ”     |
meal.feature
Feature: Meal πŸ› feature

  Background:
    Given input topic
      | topic       | alias          | key_type | value_type |
      | bread       | bread-in       | string   | string     |
      | vegetable   | vegetable-in   | string   | string     |
      | meat        | meat-in        | string   | string     |
      | side-dishes | side-dishes-in | string   | string     |
    And output topic
      | topic | alias    | key_type | value_type | readTimeoutInSecond |
      | meal  | meal-out | string   | string     | 20                  |
    And var uuid = call function: uuid

  Scenario: Left Join with Left first
    When records with key and value are sent
      | topic_alias    | key        | value |
      | bread-in       | 🀀_${uuid} | 🍞    |
      | vegetable-in   | 🀀_${uuid} | πŸ…    |
      | meat-in        | 🀀_${uuid} | πŸ₯©    |
      | side-dishes-in | 🀀_${uuid} | πŸ₯”πŸΊ  |
    Then expected records
      | topic_alias | key        | value |
      | meal-out    | 🀀_${uuid} | notif |
      | meal-out    | 🀀_${uuid} | order |
    And assert notif $ == "πŸ”"
    And assert order $ == "πŸ›(πŸ” + 🍟🍺)"

  Scenario: Left Join with Right first
    When records with key and value are sent
      | topic_alias    | key        | value |
      | side-dishes-in | 🀀_${uuid} | πŸ₯”πŸ·  |
      | bread-in       | 🀀_${uuid} | 🍞    |
      | vegetable-in   | 🀀_${uuid} | πŸ…    |
      | meat-in        | 🀀_${uuid} | πŸ₯©    |
    Then expected records
      | topic_alias | key        | value |
      | meal-out    | 🀀_${uuid} | order |
    And assert order $ == "πŸ›(πŸ” + 🍟🍷)"